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Effective  operational  test  and  evaluation  (OT&E)  is  an  essential  part  of  successful  systems  and  software  engineering.  But 
increased  program  dependencies,  network-centric  operations,  and  growing  interoperability  requirements  have  greatly  compli¬ 
cated  test  and  evaluation.  This  article  examines  the  polity,  process,  and  practice  of  the  Air  Force  (AF)  test  and  evaluation 
programs,  such  as  Force  Ftevelopment  Evaluations  (F DEs),  particularly  during  the  sustainment  of  systems.  Several  obser¬ 
vations  are  made  regarding  the  current  process  and five  areas  are  emphasised for  improvement. 


An  increasing  challenge  is  facing  the 
OT&E  community  when  operating 
across  multiple  weapon  systems  at  various 
stages  of  development.  After  studying  the 
policy,  process,  and  practice  associated 
with  an  AF  Major  Command’s  (MAJ- 
COM’s)  OT&E  program,  this  article  con¬ 
cludes  that  the  AF,  while  espousing  a  test¬ 
ing  philosophy  of  seamless  verification,  still 
needs  to  transition  to  a  more  integrated 
system  of  systems  (SoS)  approach  to  plan¬ 
ning  and  executing  OT&E.  Too  often,  a 
fielding  decision  for  a  single  system  modi¬ 
fication  is  the  goal  for  smaller  test  events. 
This  system-centric  focus  can  be  mis¬ 
placed.  Indeed,  with  the  dawning  of  net- 
work-centric  operations,  the  SoS  impera¬ 
tive  is  even  greater  for  the  successful  inte¬ 
gration,  test,  and  evaluation  of  warfight¬ 
ing  capabilities.  This  article  highlights  sev¬ 
eral  observations  and  offers  some  areas 
for  process  improvement. 

To  confirm  these  challenges,  this 
analysis  focused  on  a  geographically  dis¬ 
persed  network  of  ground  stations  that 
work  with  AF  and  DoD  surveillance  and 
reconnaissance  (S&R)  platforms  to  pro¬ 
vide  data,  information,  and  knowledge 
services  for  die  joint  commander  and 
forces  in  the  field.  A  decade  ago,  these 
S&R  platforms  and  their  attending  ground 
stations  existed  in  isolation.  Since  then, 
however,  operational  necessities  and  tech¬ 
nological  opportunities  have  birthed  a  sys¬ 
tem  of  increasingly  interdependent  hard¬ 
ware  and  software  systems,  spanning  sen¬ 
sors,  platforms,  data  links,  communication 
networks,  and  software-intensive  ground 
processing  resources.  The  evolution  of 
this  SoS  has  produced  remarkable 
advances  in  the  warfighting  capability,  but 
dais  integration  has  also  created  a  host  of 
systems  engineering  (SE)  and  enterprise 
management  challenges,  such  as  OT&E 
planning  and  execution. 

The  SoS  Challenge 

The  very  nature  of  an  SoS  makes  the 
enterprise  management  of  traditionally 
system-centric  support  processes,  such  as 


OT&E,  difficult.  As  Mark  Maier  points 
out,  even  though  an  SoS  operates  syner- 
gistically,  systems  in  an  SoS  can  operate 
and  are  managed  independendy  [1],  This 
is  a  premise  that  is  expected  to  continue 
into  the  foreseeable  future;  dierefore,  a 
single  organization  or  program  manager 
will  not  suddenly  take  complete  manager¬ 
ial  control  of  all  systems  within  the  applic¬ 
able  SoS.  One  reason  is  that  a  system  may 
participate  in  multiple  mission  direads 
interacting  widi  a  variety  of  joint  organi¬ 
zations  and  weapon  systems.  The 
“Systems  Engineering  Guide  for  Systems 
of  Systems”  recognizes  that  an  SoS  is  usu¬ 
ally  not  born  of  a  single  development 
effort  but  emerges  as  complex  combina¬ 
tions  of  newly  acquired  and  legacy  sys¬ 
tems — each  widi  their  own  management, 
operations,  and  support  communities — 
diat  evolve  over  time  [2],  Annette  Krygiel 
notes  that  the  purpose  and  capabilities  of 
an  SoS  change  as  functions  are  added, 
removed,  and  modified  [3] .  Compounding 
die  complexity  of  SoS  SE,  Pin  Chen  and 
Jennie  Clodiier  observe  that  component 
systems  in  an  SoS  are  often  systems  of 
systems  diemselves  [4],  All  of  this  compli¬ 
cates  the  current  test  and  evaluation 
approaches. 

Despite  these  challenges,  operational 
demands  are  forcing  the  AF  and  DoD  to 
co-evolve  historically  system-centric 
processes  like  OT&E  to  support  the 
development  of  SoS  and  net-centric  capa¬ 
bilities  [5],  Therefore,  to  better  understand 
die  need  for  and  obstacles  to  diis  co-evo- 
lution,  diis  study  focused  on  a  particular 
OT&E  process  and  die  extent  to  which  it 
supports  an  SoS  approach.  The  OT&E 
process  chosen,  called  an  FDE,  is  man¬ 
aged  at  the  MAJCOM  level  to  make  field¬ 
ing  decisions  for  operational  weapon  sys¬ 
tems  as  incremental  upgrades  are  made 
during  sustainment.  It  should  be  noted 
diat  an  FDE  is  one  of  several  types  of 
OT&E  called  out  in  AF  Instruction  (AFI) 
99-3,  Capability  Based  Test  and 
Evaluation.  Odiers  include  Initial  Opera¬ 
tional  Test  and  Evaluation,  Qualification 


Operational  Test  and  Evaluation,  Follow- 
on  Operational  Test  and  Evaluation, 
Tactics  Development  and  Evaluation,  the 
Weapons  System  Evaluation  Program, 
Operational  Utility  Evaluation,  Opera¬ 
tional  Assessments,  and  Early  Operational 
Assessments. 

MAJCOMs  conduct  FDEs  for  pro¬ 
grams  requiring  full-rate  production  or 
fielding  decisions  if  the  AF  Operational 
Test  Center  chooses  not  to  conduct 
OT&E.  This  is  typically  true  for 
Acquisition  Category  III  programs  or 
maintenance  modifications.  After  a  system 
has  been  fielded  and  has  entered  the  sus¬ 
tainment  phase  of  its  life  cycle,  the  prima¬ 
ry  type  of  test  and  evaluation  used  to  ver¬ 
ify  and  validate  smaller  system  upgrades  is 
the  FDE.  As  stated  in  the  Air  Combat 
Command  instruction,  the  focus  of  FDE 
is  a  subset  of  OT&E.  FDEs  are  primarily 
concerned  with  sustainment,  pre-planned 
product  improvement,  as  well  as  tactics, 
techniques,  and  procedures  development. 
The  objective  is  to  demonstrate  the  oper¬ 
ational  effectiveness  and  suitability  of  a 
system  as  evolutionary  upgrades  are  made 
to  sustain  its  relevance  to  the  warfighter. 
In  die  Air  Combat  Command  (ACC)  FDE 
process,  ACC  Test  Centers  (e.g.,  die  AF 
Warfare  Center,  the  AF  Information 
Operations  Center,  and  the  Air  National 
Guard  AF  Reserve  Test  Center)  are 
responsible  for  planning  and  executing 
FDEs. 

Next,  die  OT&E  process  is  examined 
in  die  context  of  its  governing  law  and 
policy.  Then,  a  case  study  is  documented 
which  focuses  on  a  particular  test  event 
that  involved  a  networked  ground  system 
and  one  of  its  airborne  partners. 

Observations  From  the  Field 
Study 

From  Congress,  direction  for  OT&E 
flows  down  from  four  sections  of  Title  10 
of  the  U.S.  Code:  Director  of  OT&E; 
Survivability  Testing  and  Lethality  Testing 
Required  Before  Full-Scale  Production; 
OT&E  of  Defense  Acquisition  Programs; 
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System  Life  Cycle  Testing  (horizontal) 


DT&E  =  Developmental  Testing 

Figure  1 :  System  Integration  During  Interoperability  ’Testing 


and  Low- Rate  Initial  Production  of  New 
Systems.  The  DoD  then  implements  the 
policies  into  directives,  instructions,  and 
regulations.  The  AF  further  clarifies  its 
entire  test  and  evaluation  process  in  the 
99-series  of  departmental  instructions, 
such  as  AF  Policy  Directive  99-1,  Test  and 
Evaluation  Process,  and  AFI  99-103, 
Capabilities  Based  Test  and  Evaluation  [6]. 
This  policy  defines  the  purpose  of  the  AF 
test  and  evaluation  process  and  provides  a 
framework  for  test  activities.  It  also 
expands  on  die  two  major  types  of  tests: 
Developmental  Test  and  Evaluation  and 
OT&E.  The  AF  philosophy  clearly 
reflects  the  verification  and  validation  of 
mission-level  capabilities,  an  emphasis  on 
seamless  verification  across  the  developmen¬ 
tal  and  operational  test  activities,  the  use 
of  an  integrated  test  team  (ITT)  for  test 
management,  and  the  efficient  sharing  of 
test  data  through  a  common  database. 
Finally,  AF  MAJCOMs  define  operating 
procedures,  as  in  ACC  Instruction  (ACCI) 
99-101,  ACC  Test  and  Evaluation.  This 
instruction  further  clarifies  MAJCOM 
OT&E  procedures.  Two  examples  that 
stand  out  in  ACCI  99-101  are  the 
Electronic  Project  Order,  for  tasking  orga¬ 
nizations  and  resources,  and  the  use  of  a 
yearly  Test  Priority  List.  While  extensive, 
the  test  policy  hierarchy  is  observed  to 
provide  good  vision  and  insightful  guid¬ 
ance  and  establishes  a  foundation  for 
being  able  to  handle  today’s  SoS  test  chal¬ 
lenges.  Several  observations  on  the  cur¬ 
rent  process  are  provided  in  the  following 
sections. 

Seamless  Verification  Still  Has  Seams 

Through  our  policy  analysis,  it  was  discov¬ 
ered  that  the  AF  endorses  a  capabilities- 
based  concept  of  seamless  verification, 
especially  by  mandating  the  use  of  ITTs. 
The  ITT  consists  of  a  cross-functional 
group  of  empowered  representatives 
from  multiple  disciplines  and  organiza¬ 
tions  and  is  co-chaired  by  operational 
testers  and  the  program  manager.  The 
problem  lies  in  the  meaning  of  integrated. 
While  DoD  policy  includes  both  the  hori¬ 
zontal  integration  of  test  and  evaluation 
throughout  a  system’s  life  cycle  and  the 
vertical  integration  of  systems  under  inter¬ 
operability  testing  (as  shown  in  Figure  1), 
the  AF  sees  integrated  testing  almost 
exclusively  in  life-cycle  terms  [6] .  In  addi¬ 
tion,  AF  policy  also  mandates  the  use  of 
open,  shared  databases  for  managing  test 
information.  This  integrated  information 
management  could  be  how  the  acquisition 
and  test  communities  could  begin  to 
bridge  the  vertical  seams. 

Thus,  seamless  verification  still  has 


seams  separating  test  activities  among 
interdependent  weapons  systems.  AF  pol¬ 
icy  does  not  mandate  organizational  struc¬ 
tures  and  management  processes  that  help 
OT&E  organizations  conduct  their  testing 
activities  in  an  SoS  framework.  However, 
the  prevalence  of  systems  of  systems  and 
the  evolution  toward  net-centricity  de¬ 
mands  test  processes  that  are  both  inte¬ 
grated  throughout  the  life  cycle  of  a  single 
weapon  system  and  integrated  across 
entire  sets  of  operational  capability. 

SoS  Approach  Not  Built  In 

It  was  found  that  the  FDE  process  is  flex¬ 
ible  in  that  it  can  be  applied  to  both  sys¬ 
tem-centric  and  capabilities-based  SoS  test 
events.  Even  so,  it  does  not  include  steps 
to  intentionally  evaluate  SoS  capabilities 
rather  than  individual  systems.  Rather,  it 
relies  on  the  insight  and  foresight  of  the 
MAJCOM  staff  and  the  test  center  orga¬ 
nizations  to  properly  scope  events  to 
approximately  demonstrate  full  warfight¬ 
ing  capabilities. 

Indeed,  the  test  project  manager,  gen¬ 
erally  appointed  from  one  of  the 
MAJCOM’s  test  center  organizations,  is 
the  central  figure  in  defining  the  scope  of 
the  test.  Whether  determining  the  compo¬ 
sition  of  the  planning  team,  developing 
test  objectives,  requesting  additional  sup¬ 
port  for  testing,  or  developing  the  test 
plan,  the  extent  to  which  FDE  demon¬ 
strates  the  operational  capability  of  an  SoS 
depends  largely  on  the  vision  and  initiative 
of  the  individual  project  manager.  The 
MAJCOM’s  process  does  not  include 
functions  that  obligate  a  project  manager 
to  scope  a  test  at  the  SoS  level. 

Increasing  Load  on  OT&E 

The  studied  MAJCOM  has  experienced  a 
dramatic  increase  in  OT&E  requirements 
(from  around  200  just  five  years  ago  to 


around  300  today)  and  the  number  of 
short-notice  or  out-of-cycle  requirements 
(from  around  10  percent  of  the  total  num¬ 
ber  of  test  events  five  years  ago  to  approx¬ 
imately  40  percent  today).  The  war  on  ter¬ 
ror  has  certainly  contributed  to  both  the 
number  and  urgency  of  OT&E  require¬ 
ments,  but  one  subject  matter  expert  inter¬ 
viewed  believes  the  increased  number  of 
acquisition  spirals  and  increments  along 
with  the  introduction  of  non-traditional 
software-intensive  weapon  systems — such 
as  the  networked  ground  system  in  our 
case  study — has  led  to  a  more  expansive, 
dynamic,  and  complex  environment  for 
MAJCOM-led  testing. 

Experts  and  senior  leaders  have  argued 
that  tire  growing  interdependence  of  sys¬ 
tems  organized  in  net-centric  architectures 
will  exacerbate  the  increased  load  (i.e.,  die 
number,  complexity,  tempo,  and  expense 
of  test  events),  calling  it  exponential 
growth.  With  testing  resources  either 
remaining  static  or  decreasing  over  time, 
this  increased  load  will  force  the  MAJ¬ 
COM — as  well  as  the  broader  test  com¬ 
munity — to  develop  new  methods  for  test¬ 
ing  and  evaluating  net-centric  capabilities. 

System-Centric  Approach  Breaks 
Down 

Our  case  study  starts  with  an  FDE  for  a 
modified  sensor  on  board  an  airborne 
platform.  This  sensor  modification  was 
needed  to  support  interoperability  with  a 
new  data  link  architecture,  and  it  included 
minor  software  upgrades  to  networked 
ground  stations.  The  MAJCOM  assigned 
the  event  to  its  organization  responsible 
for  testing  modifications  to  airborne  plat¬ 
forms.  Understanding  the  organic  rela¬ 
tionship  between  the  platform  and  the 
ground  system,  test  planners  knew  they 
needed  to  demonstrate  the  end-to-end 
interoperability  of  four  interdependent 
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systems:  the  sensor,  airborne  platform, 
data  link,  and  ground  station.  Yet,  the  test 
was  not  planned  as  a  demonstration  of  die 
operational  effectiveness  and  suitability  of 
an  SoS  capability,  but  as  a  validation  of  die 
single-sensor  system  with  die  support  of 
diese  other  contributing  systems.  While 
this  may  seem  like  a  subtle  distinction,  it 
led  to  a  variety  of  coordination  and  com¬ 
munication  breakdowns  throughout  test 
planning.  In  retrospect,  the  FDE  and  sub¬ 
sequent  fielding  decision  should  have  been 
for  the  combined  SoS,  made  up  of  the  sen¬ 
sor,  airborne  platform,  data  link,  and 
ground  stations,  which  would  have  neces¬ 
sarily  involved  a  broader  cross-section  of 
stakeholders  from  the  genesis  of  die  test 
event.  When  SoS  thinking  is  not  baked  into 
die  overall  test  and  evaluation  process, 
even  highly  interdependent  systems  will 
have  difficulty  coordinating  test  events 
that  are  effective  and  relevant  for  the  con¬ 
stituent  systems  and  the  SoS  as  a  whole. 

Recommendations  for 
Net-Centric  OT&E 

Though  important  efforts  have  been 
made  at  all  levels  to  promote  SoS-level 
testing,  the  default  focus  of  testing  is  still 
on  individual  systems  as  opposed  to  whole 
capabilities.  As  net-centric  operations 
mature,  this  approach  will  have  to  change. 
Indeed,  as  our  field  study  indicates,  the 
DoD  is  already  feeling  die  pressure  that 
was  predicted  nine  years  ago: 

Testing  systems  will  become  far 
more  complex  since  the  focus  will 
not  be  on  the  performance  of  indi¬ 
vidual  systems,  but  on  the  perfor¬ 
mance  of  federations  of  systems.  [7] 

At  the  National  Defense  Industrial 
Association  Test  and  Evaluation  Summit 
in  2004,  DoD  officials  elaborated  on  the 
net-centric  challenges  to  traditional,  sys¬ 
tem-centric  testing  [8]: 

•  The  shifting  focus  from  platforms  to 
capabilities  and  SoS  solutions. 

•  The  increasing  complexity  of  systems 
combined  with  increasing  interdepen¬ 
dencies  among  systems  of  systems. 

•  The  increasing  operational  demand  for 
broader  and  deeper  integration  among 
disparate  systems. 

•  The  exponential  growth  in  functional 
and  physical  interfaces  introduced  by 
the  proliferation  of  network  partici¬ 
pants  (both  newly  developed  and 
legacy). 

•  The  increased  requirements  for  test 
and  evaluation  initiated  by  the  evolu¬ 
tionary  acquisition  philosophy  of 


build-a-little,  test-a-little. 

As  the  heralds  of  net-centricity 
emphasize,  the  DoD’s  transition  from 
Industrial  Age  (platform-centric)  to 
Information  Age  (net-centric)  operations 
must  include  a  co-evolution  of  supporting 
processes.  Testing  is  one  of  those  sup¬ 
porting  processes  that  must  co-evolve 
with  technology.  The  following  recom¬ 
mendations  are  believed  to  best  improve 
MAJCOM-level  OT&E,  but  should  be 
extensible  to  the  AF  and  DoD  OT&E. 
These  can  be  considered  attributes  of  a 
future  process;  while  not  meant  to  be 
exhaustive,  they  provide  a  starting  point 
for  continuing  research  on  how  to  evolve 
today’s  process  to  complement  a  more 
interoperable  net-centric  environment. 

Scope  OT&E  Events  at  the  SoS  Level 

This  article  advocates  an  SoS  (instead  of  a 
system-centric)  approach  to  OT&E. 
However,  die  question  arises  of  how  to 
appropriately  scope  the  boundaries  of  SoS 
test  events.  This  is  where  the  DoD  AF  and 
AF  Enterprise  Architecture  (EA)  could 
offer  practical  help.  As  the  use  of  EAs 
continues  maturing,  they  will  provide 
effective  models  for  assessing  how 
weapon  systems  can  and  should  interoper¬ 
ate  in  order  to  provide  warfighting  capa¬ 
bilities  to  the  joint  commander.  These 
models  will  help  planners  define  the 
boundaries  of  an  SoS,  and  they  will  docu¬ 
ment  what  the  MITRE  Corporation’s 
Prem  Jain  calls  a  mission  thread-. 

A  precise,  objective,  description  of 
an  important  task  ...  a  time- 
ordered  operational  event  diagram 
that  captures  discrete,  definable, 
interactions  among  human  opera¬ 
tors  and/or  technological  compo¬ 
nents.  [9] 

Jain  argues  that  these  mission  threads  will 
support  modeling  and  simulation  (M&S) 
activities  for  net-centric  test  and  evalua¬ 
tion.  In  die  same  way  the  AF  uses  high- 
fidelity  simulators  to  slash  die  costs  asso¬ 
ciated  with  training  its  pilots,  the  test  com¬ 
munity  could  use  mission  thread-based 
M&S  to  validate  SoS  and  net-centric  capa¬ 
bilities  at  a  fraction  of  die  cost,  time,  and 
operational  impact  incurred  by  live,  end- 
to-end  tests. 

Validate  SoS  Interoperability 

By  advocating  SoS  testing,  an  endless  web 
of  end-to-end  interoperability  tests  is  not 
envisioned  with  every  other  possible 
weapon  system  and  every  configuration. 
Rather,  changes  to  individual  weapon  sys¬ 
tems  should  be  evaluated  according  to 


net-readiness  criteria  to  validate  their 
interoperability  with  the  rest  of  the  SoS  or 
net-centric  enterprise.  This  does  not  mean 
just  checking  to  see  if  a  modified  weapon 
system  is  IP-enabled.  Net-readiness  is  a 
comprehensive  concept  that  implies  inter¬ 
operability  at  many  layers  of  the  commu¬ 
nications  hierarchy  and  beyond:  physical, 
logical,  syntactic,  and  semantic  [2],  For 
nearly  30  years,  both  government  and 
industry  have  actively  explored  research 
on  interoperability  measurement  with  the 
goal  of  creating  a  straightforward  way  of 
measuring,  reporting,  and  then  improving 
the  interoperability  of  complex  networks 
of  people,  equipment,  processes,  and 
organizations.  Researchers  have  used 
more  than  30  definitions  of  interoperabil¬ 
ity  and  have  documented  more  than  60 
distinct  pipes  of  interoperability,  numer¬ 
ous  interoperability  attributes,  and  14 
foundational  interoperability  measure¬ 
ment  models  and  methodologies  [10], 

For  SoS  testing,  a  set  of  net-readiness 
objectives  (see  Table  1)  based  on  the 
DoD’s  Net-Centric  Data  Strategy  [11]  is 
proposed.  Incorporating  these  objectives 
into  SoS  events  would  ensure  that  modi¬ 
fied  systems  continue  to  conform — at 
their  interface  with  die  network — to  the 
convergence  protocols  specified  in  the 
net-centric  architecture.  Instead  of  evalu¬ 
ating  all  end-to-end  relationships  to  vali¬ 
date  interoperability  within  the  SoS,  a  test 
event  could  confirm  the  integrity  of  the 
SoS  simply  by  demonstrating  the  modified 
system’s  adherence  to  the  network’s  con¬ 
vergence  protocols.  This  technique  would 
greatly  reduce  the  test  load  on  OT&E 
organizations  while  simultaneously  allow¬ 
ing  them  to  conduct  evaluations  focused 
on  the  operational  effectiveness  of  the 
net-centric  SoS,  as  opposed  to  the  individ¬ 
ual  systems  within  that  SoS. 

Prioritize  FDEs  According  to 
Operational  Risk 

Although  a  simulation  and  net-readiness 
demonstration  at  the  network  interface 
will  help  mitigate  the  test  load  associated 
with  SoS  and  net-centric  operations,  deci¬ 
sion  makers  will  still  expect  a  certain  level 
of  live,  end-to-end  testing  in  realistic  sce¬ 
narios  to  validate  higher-risk  capabilities. 
There  will  always  be  too  much  to  test, 
forcing  the  operational  and  test  communi¬ 
ties  to  develop  a  reasonable  means  of  pri¬ 
oritizing  test  events.  Complicating  this 
issue  is  a  fundamental  property  of  net- 
centric  operations:  New  transactions, 
interdependencies,  missions,  and  capabili¬ 
ties  as  additional  (even  unanticipated)  sen¬ 
sor,  shooter,  and  command  and  control 
nodes  join  the  network.  The  very  nature 
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of  net-centric  operations  implies  that  the 
DoD  will  never  completely  anticipate  all 
of  the  relevant  operational  nodes  and  pre¬ 
cisely  how  those  nodes  will  interoperate  to 
accomplish  a  mission. 

Thus,  the  crucial  criterion  for  prioritiz¬ 
ing  and  scoping  SoS  OT&E  events  is 
operational  risk.  For  low-risk  development 
or  sustainment  efforts,  operational  deci¬ 
sion  makers  may  need  to  be  satisfied  with 
developmental  test  results  validated  in  an 
M&S-based  operational  test.  For  medium- 
risk  projects,  testers  may  use  a  synthetic 
test  strategy  that  employs  a  small  number 
of  distributed  operational  events  in  an 
M&S  framework.  The  test  and  evaluation 
community  may  need  to  reserve  tradition¬ 
al  end-to-end  events  for  the  highest  risk 
efforts.  The  key  will  be  for  operational 
decision  makers  to  set  an  appropriate  risk 
threshold  for  each  development  or  sus¬ 
tainment  program  and  seek  the  best  value 
test  option  in  terms  of  time,  money,  and 
testing/operational  resources  to  achieve 
that  threshold. 

Focus  on  Interfaces 

Without  trivializing  the  complexities  of  an 
SoS,  perhaps  more  emphasis  may  have  to 
be  placed  on  the  definition,  development, 
and  test  and  evaluation  of  interfaces. 
Interfaces  and  interface  management  have 
always  been  an  important  aspect  of  SE. 
Maier  and  Rechtin  state  this  design  heuris¬ 
tic:  “The  greatest  leverage  in  system  archi¬ 
tecting  is  at  the  interfaces  . . .  the  greatest 
dangers  are  also  at  the  interfaces”  [12], 
According  to  the  “Defense  Acquisition 
Guidebook,”  this  heuristic  can  be  an  inter¬ 
face  that  is: 

The  [logical],  functional,  and  phys¬ 
ical  characteristics  required  to  exist 
at  a  common  boundary  or  connec¬ 
tion  between  persons,  between  sys¬ 
tems,  or  between  persons  and  sys¬ 
tems.  [13] 

During  an  interface  control  document 
(ICD)  review  of  one  space  program,  we 
examined  the  impact  of  interface  design 
and  development.  Over  a  three-year  peri¬ 
od  following  contract  award,  596  program 
engineering  items  were  examined.  These 
items  included:  requirements  changes, 
specification  updates  and  clarifications, 
ICD  changes,  SE  management  docu¬ 
ments,  baseline  schedule  changes,  test 
plans,  and  proposed  risk-mitigation 
actions;  ICD-related  actions  comprised 
190  (or  one-third)  of  the  total  number  of 
actions.  A  second  aspect  of  interface  man¬ 
agement  was  in  contract  costs.  From  a  set 
of  77  contractual  modifications  after  crit¬ 


Objective 

Description 

Visible 

Users  and  applications  can  discover  the  existence  of  data  assets 
through  catalogs,  registries,  and  other  search  services.  All  data 
assets  (intelligence,  non-intelligence,  raw,  and  processed)  are 
advertised  or  made  visible  by  providing  metadata,  which  describes 
the  asset. 

Accessible 

Users  and  applications  post  data  to  a  shared  space.  Posting  data 
implies  that:  (1 )  descriptive  information  about  the  asset  (metadata) 
has  been  provided  to  a  catalog  that  is  visible  to  the  enterprise  and 
(2)  the  data  is  stored  such  that  users  and  applications  in  the 
enterprise  can  access  it.  Data  assets  are  made  available  to  any 
user  or  application  except  when  limited  by  policy,  regulation,  or 
security. 

Understandable 

Users  and  applications  can  comprehend  the  data,  both  structurally 
and  semantically,  and  readily  determine  how  the  data  may  be  used 
for  their  specific  needs. 

Trusted 

Users  and  applications  can  determine  and  assess  the  authority  of 
the  source  because  the  pedigree,  security  level,  and  access 
control  level  of  each  data  asset  is  known  and  available. 

Agile 

Many-to-many  exchanges  of  data  occur  between  systems,  through 
interfaces  that  are  sometimes  predefined  or  sometimes 
unanticipated.  Metadata  is  available  to  allow  mediation  or 
translation  of  data  between  interfaces,  as  needed. 

Responsive 

Perspectives  of  users,  whether  data  consumers  or  data  producers, 
are  incorporated  into  data  approaches  via  continual  feedback  to 
ensure  satisfaction. 

Table  1:  A  Template  for  Net-Readiness  Testing  Objectives 


ical  design  review,  43  (nearly  56  percent) 
were  in  some  way  related  to  interfaces 
either  through  studies,  ICD  updates,  or 
implementations  of  necessary  require¬ 
ment  changes.  More  interestingly,  ICD- 
related  issues  resulted  in  nearly  $31.5  mil¬ 
lion  (or  44  percent)  of  the  cost  impact  to 
dais  program.  Although  this  is  just  one 
example,  it  is  an  indication  of  interface 
management  challenges  during  develop¬ 
ment  and  clearly  represent  an  area  that  will 
require  similar  effort  during  OT&E. 

Unfortunately,  interface  management 
alone  may  be  insufficient  to  understand 
the  complexity  within  an  SoS.  For  net¬ 
work-centric  information  systems,  one 
must  examine  the  internal  properties  and 
behaviors  of  the  logically  connected  sys¬ 
tems  and  their  users  who  find,  fuse,  mod¬ 
ify,  and  ultimately  use  shared  data. 

Employ  Integration  Environments 

The  heavy  use  of  M&S  and  synthetic  test¬ 
ing  to  supplement  traditional  test  and  eval¬ 
uation  presupposes  the  use  of  integration 
environments  to  build  SoS  and  net-centric 
operational  capabilities.  Annette  Krygiel 
calls  an  integration  environment: 

...  a  concept,  not  an  organization. 

It  is  the  environment  of  people, 
processes,  and  infrastructure  used 
by  a  team  consisting  of  acquisition 
and  operational  personnel  to  man¬ 
age  the  integration  before  the 
product  is  deployed  for  an  opera¬ 


tion  or  an  experiment  and  to  sus¬ 
tain  it  afterward.  [3] 

Thus,  an  integration  environment  is  a 
concept  that  transcends  not  just  OT&E 
but  is  an  essential  SE  infrastructure  for  the 
early  and  continuous  integration  of  testing 
efforts  in  SoS  development  and  sustain¬ 
ment.  Thus,  an  integration  environment  is 
critical  in  squeezing  the  most  overall  value 
from  OT&E  and  in  reducing  the  amount 
of  effort  required  late  in  the  development 
cycle  [14], 

Clearly,  integrated  databases  open  to 
all  test  and  evaluation  stakeholders  are  just 
the  tip  of  the  iceberg  in  terms  of  the  tools 
needed  to  achieve  seamless  verification. 
Employing  integration  environments 
would  allow  test  teams  to  achieve  synergy 
throughout  the  life  cycle  of  component 
systems  and  across  the  networked  SoS.  It 
would  allow  early,  comprehensive,  and 
ubiquitous  test  and  evaluation  throughout 
the  development  of  SoS  capabilities, 
reducing  the  test  load  on  test  center  orga¬ 
nizations  and  giving  them  the  freedom  to 
focus  on  adding  value  where  it  counts  for 
MAJCOM  decision  makers:  in  reducing 
the  operational  risk  of  fielding  and 
employing  warfighting  capabilities. 

Summary 

This  field  study  of  the  policy,  process,  and 
practice  of  FDE  concludes  that  the  testing 
community  must  shift  from  system-centric 
testing  to  a  more  SoS  approach.  The 
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DoD’s  ongoing  transformation  to  net-cen¬ 
tric  operations  makes  die  co-evolution  of 
SoS  test  and  evaluation  an  even  greater 
imperative.  While  the  five  areas  for 
improvement  were  derived  from  MAJ- 
COM  FDE  practice,  they  reflect  similar 
concepts  for  more  formal  OT&E. 
Improving  die  operational  realism  of  net¬ 
work-centric  environments,  ensuring  time¬ 
ly  performance  of  operational  informa¬ 
tion,  and  facilitating  adequate  test  and  eval¬ 
uation  resources  continue  to  be  priorities 
of  die  OT&E  director  [15].  Likewise,  AF 
OT&E  continues  to  be  challenged  widi 
SoS  test  planning  and  execution.  The  soft¬ 
ware  engineering  community  must  contin¬ 
ue  to  design  and  test  capabilities  widi  an 
SoS  focus  and  develop  advanced  modeling 
and  simulation  capabilities  to  enable 
affordable  SoS  testing  in  a  net-centric  envi¬ 
ronment.  Likewise,  die  test  community, 
while  emphasizing  seamless  verification, 
needs  to  make  continued  progress  in  capa- 
bilities-based  interoperability  testing  across 
information-intensive  SoS.^ 
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